Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Elastic Search 5 Indexing Performance Issue with product mapper #25452

Conversation

behnamshayani
Copy link
Contributor

@behnamshayani behnamshayani commented Nov 3, 2019

Description (*)

This PR fixes performance issue with huge catalog that have attributes that can have thousands of different options (in our case we have 23000 option for one and 3000-4000 for some others)
The original code for each product, iterates over all possible options to find corresponding label for a product's possible options values (so if a product has 10 option values out of 23000 options, the original code does 22990 extra loops and there is also use of in_array in each of them)
The fix aims to build the attribute option array by the option values as key, so it can only iterate the product's option values (10 items) and use isset instead of in_array

Fixed Issues (if relevant)

  1. Elastic Search 5 Indexing Performance Issue #20966: Elastic Search 5 Indexing Performance Issue

Manual testing scenarios (*)

  1. test with a huge catalog with million of products and attributes with more than 10000 options
  2. reindex catalogsearch_fulltext
  3. the products should be indexes in elastic search and be filterablbe

Questions or comments

On a catalog with 1.2 Million product , an attribute with 23000 options and a couple with 3000-4000 and 4 websites the catalogsearch_fulltext took 12hours before the fix and 45min on local devbox after the fix.

Contribution checklist (*)

  • Pull request has a meaningful description of its purpose
  • All commits are accompanied by meaningful commit messages
  • All new or changed code is covered with unit/integration tests (if applicable)
  • All automated tests passed successfully (all builds are green)

@behnamshayani behnamshayani requested a review from kokoc as a code owner November 3, 2019 20:02
@m2-assistant
Copy link

m2-assistant bot commented Nov 3, 2019

Hi @behnamshayani. Thank you for your contribution
Here is some useful tips how you can test your changes using Magento test environment.
Add the comment under your pull request to deploy test or vanilla Magento instance:

  • @magento give me test instance - deploy test instance based on PR changes
  • @magento give me 2.3-develop instance - deploy vanilla Magento instance

For more details, please, review the Magento Contributor Guide documentation.

@ihor-sviziev ihor-sviziev added the Auto-Tests: Not Required Changes in Pull Request does not require coverage by auto-tests label Nov 4, 2019
@magento-engcom-team
Copy link
Contributor

Hi @ihor-sviziev, thank you for the review.
ENGCOM-6232 has been created to process this Pull Request

@magento-engcom-team
Copy link
Contributor

@behnamshayani thank you for contributing. Please accept Community Contributors team invitation here to gain extended permissions for this repository.

@engcom-Alfa
Copy link
Contributor

✔️ QA Passed

Should be created and run performance tests

@m2-assistant
Copy link

m2-assistant bot commented Jan 8, 2020

Hi @behnamshayani, thank you for your contribution!
Please, complete Contribution Survey, it will take less than a minute.
Your feedback will help us to improve contribution process.

@sdzhepa sdzhepa mentioned this pull request May 9, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants